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Abstract 

Background: Scientific knowledge is in constant change. The flow of new information requires a frequent 
re-evaluation of the available research results. Clinical practice guidelines (CPGs) are not exempted from this 
phenomenon and need to be kept updated to maintain the validity of their recommendations. The objective of 
our review is to systematically identify, describe and assess strategies for monitoring and updating CPGs. 

Study design and setting: We conducted a systematic review of studies evaluating one or more methods of 
updating (with or without monitoring) CPGs or recommendations. We searched MEDLINE (PubMed) and The 
Cochrane Methodology Register (The Cochrane Library) from 1966 to June 2012. Additionally, we hand-searched 
reference lists of the included studies and the Guidelines International Network book of abstracts. If necessary, we 
contacted study authors to obtain additional information. 

Results: We included a total of eight studies. Four evaluated if CPGs were out of date, three updated CPGs, and 
one continuously monitored and updated CPGs. The most detailed reported phase of the process was the 
identification of new evidence. As opposed to studies updating guidelines, studies evaluating if CPGs were out of 
date applied restricted searches. Only one study compared a restricted versus an exhaustive search suggesting that 
a restricted search is sufficient to assess recommendations' Validity. One study analyzed the survival time of CPGs 
and suggested that these should be reassessed every three years. 

Conclusions: There is limited evidence about the optimal strategies for monitoring and updating clinical practice 
guidelines. A restricted search is likely to be sufficient to monitor new evidence and assess the need to update, 
however, more information is needed about the timing and type of search. Only the exhaustive search strategy has 
been assessed for the update of CPGs. The development and evaluation of more efficient strategies is needed to 
improve the timeliness and reduce the burden of maintaining the validity of CPGs. 

Keywords: Clinical practice guidelines, Diffusion of innovation, Evidence-based medicine, Information storage and 
retrieval, Methodology, Updating, Implementation science, Dissemination and implementation, Knowledge translation 



Background Shekelle et al. evaluated the validity of a cohort of 

Scientific knowledge is in constant change, and new CPGs [3]. Survival analysis indicated that 90% of CPGs 

information requires frequent assessment to determine were still valid in 3.6 years, but 50% were out of date in 

whether it changes the knowledge base [1]. A clinical 5.8 years [3]. Based on these results, most methodo- 

practice guideline (CPG) may be considered out of date logical handbooks for the development of CPGs propose 

if it does not include all recent, valid, and relevant evi- three years as a reasonable time frame to update their 

dence or does not reflect current clinicians' experience guidelines [1,4]. 

and patients' values and preferences [2], CPGs, hence, In 2007, Moher et al. conducted a study about when 

need to be updated regularly to remain valid. and how to update systematic reviews [5] . Although not 

included in the objectives, the authors identified and 
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updating CPGs, as opposed to systematic reviews, is well 
established. Nevertheless, a recent international survey 
showed high variability and a lack of standardization in 
the CPGs updating processes [6]. 

A few studies have evaluated different strategies for 
the CPGs updating process [3,7,8], however, no system- 
atic reviews have been conducted about this topic. We 
therefore undertook a systematic review of the studies 
that assessed strategies for monitoring and updating 
CPGs. 

Methods 

Information sources and search 

We performed a search in June 2012 in MEDLINE 
(accessed through PubMed, from 1966 onwards) and 
The Cochrane Methodology Register (accessed through 
The Cochrane Library, Issue 6 2012). We included stud- 
ies regardless of their language or publication status. 
The search strategy is available as supplementary data 
(Additional file 1). Additionally, we hand searched refer- 
ence lists of the included studies and in the Guidelines 
International Network book of abstracts (available online 
from 2009 until 2011 in http://www.g-i-n.net/). If neces- 
sary we contacted study authors to obtain additional in- 
formation. Two authors were in charge of performing all 
searches (IS, LMG). 

Eligibility criteria 

1. Type of study: We included studies evaluating one or 
more methods of updating (with or without 
monitoring) evidence-based CPGs or 
recommendations. We excluded studies that only 
reported updating methods (without actually testing 
them), methodological handbooks or CPGs updates. 
We made no restriction by health topic. 

2. Type of design: We included studies assessing 
strategies for evaluating if CPGs are out of date; for 
updating CPGs; for continuous monitoring and 
updating of CPGs (Figure 1). 

Study selection 

Two authors independently reviewed the titles and 
abstracts, as well as the full text of the selected articles 
for a more detailed evaluation and approved their final 
inclusion (LMG, IAR). Any disagreement among the 
authors was initially resolved by consensus, and if neces- 
sary, we consulted a third author (IS). 

Data extraction strategy 

Two authors independently extracted information from 
the included studies using an ad hoc form (LMG, IAR) 
that can be requested from the authors. Disagreements 
among the authors were resolved by consensus and, if 



required, by a third author (IS). We contacted study 
authors by email when more information was needed. 

We extracted the following information from each 
study: institution or guideline program and country; 
objective and design of the study; sample (selection and 
size) and health topic; time to update (number of years 
since the development of the original CPG); stages of 
the strategies; type of search (restricted or exhaustive, 
classified depending on databases consulted and types of 
studies searched); resource use (number of participants 
and time spent); search and update results; and ad- 
vantages and limitations of strategies reported by the 
authors. 

Data synthesis and presentation 

We describe included studies both individually and nar- 
ratively. We calculated the search performance of the 
strategies (as a proportion of included documents from 
all documents identified); and update performance of 
the strategies (as a proportion of updated recommenda- 
tions or CPGs from all evaluated recommendations or 
CPGs). 

Results 

Study selection 

The screening process is summarized in the flow dia- 
gram (Figure 2). We initially identified 2,923 references 
from our search strategy, and excluded 141 duplicates 
and 2,731 references after examining the title and 
abstract. We reviewed 51 full texts and we excluded 39 
references (Additional file 2). We finally included eight 
studies corresponding to 12 original references [3,7-17]. 
Three studies were only available as abstracts [9,14,15]. 
We successfully contacted two of the first abstracts' 
authors to obtain additional information of these stu- 
dies [14,15]. 

Study characteristics 

Four studies assessed strategies evaluating if CPGs were 
out of date [3,7,9,15], three studies assessed strategies 
for updating CPGs [10,14,16], and one study assessed 
a strategy for continuous monitoring and updating CPGs 
[8] (Table 1). Five studies assessed a single strategy for: 
evaluating if CPGs were out of date [3,9,15], for up- 
dating CPGs [14] or for continuous monitoring and 
updating of CPGs [8]; three studies compared two strat- 
egies: two different strategies for evaluating if CPGs were 
out of date [7] or 'de novo' development versus updating 
strategies [10,16]. Most of the studies evaluated full 
CPGs (range from one to 20 CPGs) [3,8-10,14,15], how- 
ever none of them included a random sample. The 
topics included varied widely, with cancer being the 
most frequent [8,9,16]. The time to update source docu- 
ments ranged from one to eight years. The common 



Martinez Garcia et al. Implementation Science 2012, 7:109 
http://wwwJmplementationscience.eom/content/7/1/109 



Page 3 of 10 



Evaluating if out of date 



Identifying 
retrospectively 
new evidence 



Identifying 
prospectively 
new evidence 
(monitoring) 



X 



Need for 
update? 



Updating 




Continuous monitoring and 
updating 




Figure 1 Updating Clinical Practice Guidelines Strategies. 



stages identified in the strategies were: assessment of 
need to update; scope definition of the update; working 
group composition; search strategy; selection, assess- 
ment, and synthesis of the evidence; recommendations 
update (only for updating and continuous monitoring); 
and presentation format of the updates (for strategies for 
updating and continuous monitoring) (Table 2). 

Strategies for evaluating if CPGs are out of date 

Shekelle et al. [3] developed a strategy to assess the val- 
idity of CPGs based on identified new evidence through 
restricted searches (reviews, editorials, or commentaries 
in general or specialised journals) and through a survey 
of clinical experts (Table 2). The CPGs were classified by 
the type of update required as: major update — new evi- 
dence suggests the need for new recommendations; 
minor update — new evidence supports changes to 
recommendations or refinement of existing recommen- 
dations; and the recommendations remain valid. The 
participants in this updating process were two metho- 
dologists and 146 clinical experts. The clinical experts 
participated in a survey to assess current validity of the 
guideline recommendations and to identify new evidence 
(71% of response rate). The search performance was 
2.9% (208 articles reviewed from 7,150 articles initially 
identified) (Table 3). The authors concluded that 76.5% 
of the guidelines needed to be updated (13/17 CPGs) 
(Table 3). The results of survival analysis indicated that 
90% of CPGs were still valid at 3.6 years (95% confidence 
interval (CI) 2.6 to 4.6 years), but 50% of CPGs were out 
of date at 5.8 years (95% CI 5.0 to 6.6 years). The main 
reported limitation of the study included no previous 
validation of the method of assessing obsolescence. 



Bosquet et al. [9] monitored the literature to assess the 
need for updating a CPG (Table 2). This approach 
applied an exhaustive search and evaluated the impact of 
new studies on existing guidelines (consistent, inconsist- 
ent or concern new topics) by consultation with the ori- 
ginal working group (mail and meetings). The search 
performance was 45.2% (118 references submitted to the 
working group from 261 references initially identified) 
(Table 3). 

Gardehner et al. [7] compared two search strategies to 
identify new evidence for assessing the need to update six 
topics of one CPG: a modified Shekelle et al. [3] search 
versus an exhaustive search (Table 2). The researchers 
modified in three consecutive phases the literature search 
developed by Shekelle et al. [3], mainly by eliminating 
some of the databases (omitting a general web site search 
and confining the MEDLINE search to Abridged Index 
Medicus journals). The participants were 10 methodolo- 
gists (five for each model) and 13 clinical experts. The 
clinical experts participated in a survey similar to the one 
carried out in Shekelle et al. [3] (28% response rate). The 
search performance was 2.6% for the modified Shekelle 
et al. [3] search (36 eligible studies from 1,382 citations 
initially identified) and 1.2% for the exhaustive search 
(45 eligible studies from 3,687 citations initially identified) 
(Table 3). The reported strength of the modified Shekelle 
et al. [3] method was that it offered a search strategy that 
detected fewer citations to review and identified all studies 
that triggered an update. However, the study only included 
prevention topics; the comparison of the number of 
abstracts and full texts between the two approaches was 
limited because the Shekelle et al. [3] search was an inte- 
gral part of the exhaustive search; and the experts who 
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Records identified through 
database searching 
(n=2907) 



Additional records identified 
in other sources 
(n=16) 



Total number records 

(n=2923) 



Records screened 

(n=2782) 



Full-text articles assessed 
for eligibility 

(n=51) 



Included articles 

(n=12) 



Studies included* 

(n=8) 

Shekell8 2001 [3]#; Eccles 2002 [17ft 
Bosquet 2003 [9]$ 

Gartlehner 2004 [7]#; Gartlehner 2007 [1 1]t; Voisin 2008 [12] 
Nunes 2009 [15ft§ 
Eccles 2002 [1 0] 
Newton 2006 [14ft§ 
Parmelli 2010 [16] 

Johnston 2003 [8]#; Brouwers 2001 [13J( 



Duplicates (n=141) 



Records excluded 
(n=2731) 



Full-text articles excluded 
(n=39) 

Comparison updated guidelines (n-1) 
Computer support (n=9) 
Describes development method (n-1) 
Describes update method or tool 
(n-1 5) 

Evaluates evidence sources (n-1) 
Evaluates health information 
technology (n-1) 

Methodological handbook or report 
(n=8) 

Narrative review (n-4) 
Systematic review update (n-1) 



Figure 2 Flow diagram for the identification of studies. "Ordered by type of strategy and chronology; #Main publication for the study; 
tOnly abstract available; §lnformation provided by the author. 



assessed the importance of the studies not identified were 
not blinded to the type of search approach. 

Nunes et al. [15] described a restricted search and a 
review of new CPGs to decide whether their guideline 
required an update (Table 2). The participants were three 
methodologists and two experts. The search performance 
was 28% (seven guidelines reviewed from 25 guidelines 
initially identified) (Table 3). No recommendations needed 
to be updated because identified recommendations in 
other CPGs were consistent with the existing CPG 
(Table 3). Less time was required to perform the update 
strategy than the time estimated to perform an exhaustive 
search. The need to judge the applicability of recommen- 
dations from other guidelines was one of the difficulties 
reported. 

Strategies for updating CPGs 

Eccles et al. [10] compared the updating process with 
the original development process of two CPGs (Table 2). 



In both the development and updating process new evi- 
dence was identified by exhaustive search. Recommenda- 
tions were classified as: new (if new evidence was 
identified); refined (if supplementary evidence was identi- 
fied); and unchanged (if no new evidence was identified). 
Two methodologists and 14 clinical experts participated. 
The search performance was 1.0% for each guideline 
(e.g., for angina there were 59 acceptable papers from 
5,941 citations initially identified) (Table 3). Recommenda- 
tions remained unchanged (Table 3). Among the reported 
strengths were that some members also had participated 
in the original development of the guidelines and were 
better trained and were more familiar with the literature. 
These members also identified other relevant evidence. 
The main reported limitation of the updating strategy was 
that it was as burdensome, time consuming, and costly. 

Newton et al. [14] updated an existing CPG (by updat- 
ing and expanding the original search) and adapted it to 
the local context (by adding research questions for the 
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Table 1 Characteristics of included studies* 



Author 
and year 


Institution or guideline 
program (country) 


Objective 


Design 


Sample 


Health topic 


Time to 
update 


Shekelle 
2001 [3] 


AHRQ (USA) 


Evaluating if out 
of date 


One strategy 


1 7 CPGs 


Several topics 


4-8 years 


Bosquet 
2003 [9]* 


FNCLCC (France) 


Evaluating if out 
of date 


One strategy 


1 CPG 


PET scanning 
in cancer 


NS 


Gartlehner 
2004 [7] 


RTI-UNC EPC, USPSTF, 
AHRQ (USA) 


Evaluating if out 
of date 


Two strategies 


6 topics 


Prevention topics 


6 years 


Nunes 
2009 [15]*§ 


NCGC, NICE (UK) 


Evaluating if out 
of date 


One strategy 


1 CPG 


Obesity 


3 years 


Eccles 
2002 [10] 


North of England Evidence 
Based Guideline Development 
Programme (UK) 


Updating 


Two strategies 
(development 
vs updating) 


2 CPGs 


Angina and asthma 
in adults 


4-5 years 


Newton 
2006 [14]*§ 


AHTA (Australia) 


Updating (updating 
by adapting) 


One strategy 


1 CPG 


Posttraumatic stress 
disorder 


1-3 years 


Parmelli 
2010 [16] 


ERHCA (Italy) 


Updating 


Two strategies 
(development 
vs updating) 


15 recommendations Anticancer drugs for 
breast, colorectal 
and lung cancer 


3 years 


Johnston 
2003 [8] 


CCOPGI (Canada) 


Continuously 
monitoring and updating 


One strategy 


20 CPGs 


Cancer 


NS 



"Ordered by type of strategy and chronology; +Only abstract available; information provided by the author. 

Abbreviations: AHRQ: Agency for Healthcare Research and Quality; AHTA: Adelaide Health Technology Assessment; CCOPGI: Cancer Care Ontario Practice 
Guidelines Initative; CPG: Clinical practice guideline; ERHCA: Emilia-Romagna Health Care Agency; FNCLCC: Federation Nationale des Centres de Lutte Contre le 
Cancer; NCGC: National Clinical Guidelines Centre for Acute and Chronic Conditions; NICE: National Institute for Health and Clinical Excellence; NS: Not stated; PET: 
Positron Emission Tomography; RTI-UNC EPC: RTI International-University of North Carolina Evidence-based Practice Center; UK: United Kindom; USA: United 
States of America; USPSTF: US Preventive Services Task Force; vs: Versus. 



local setting) (Table 2). The participants were nine meth- 
odologists and six experts. The search performance was 
0.2% (43 studies included from 19,423 citations initially 
identified) (Table 3). The authors spent nine months to 
update 11 research questions and to develop seven new 
research questions (Table 3). The authors acknowledged 
that they were uncertain whether the process was more 
time efficient due to the additional search needed to 
cover local adaptation issues. 

Parmelli et al. [16], similarly to Eccles et al. [10], com- 
pared an updating method with the original develop- 
ment process in a set of 15 recommendations (Table 2). 
The GRADE framework was used in both developing 
and updating recommendations [18]. For the updating 
process, new exhaustive searches were designed. The 
number of participants was not specified, but they 
described the main characteristics of its coordinator, 
methodological, and expert group. The researchers spent 
eight months to update 15 recommendations. The 
search performance was 3.5% (24 papers included from 
686 records initially screened) (Table 3). Forty per cent 
of the recommendations (6/15) were completely updated 
due to identification of new evidence, leading to a 
change in recommendation classification or to redefin- 
ition of the original clinical questions (Table 3). The 
advantages of this updating strategy, as reported by the 
authors, were better collaboration among the members 
of the updating group, and improved methodology, lead- 
ing to more relevant recommendations for clinical 



practice. On the other hand, the updating process was 
neither time nor resource saving. 

Strategies for continuous monitoring and updating of CPGs 

Johnston et al. [8] carried out a pilot study of a monitoring 
strategy in 20 CPGs (Table 2). The strategy included four 
steps: continuous and exhaustive searching; reviewing the 
new evidence; revising recommendations; and announcing 
new evidence and modified recommendations. The 
researchers used the term 'living' practice guideline, 
defined as 'integrated evidence update into the original 
report.' The information about the participants was lim- 
ited. The researchers applied this strategy during a year. 
The search performance was 23.75% (19 citations with im- 
pact on recommendations from 80 citations initially iden- 
tified) (Table 3). In 30% (6/20) of the guidelines, a change 
in their clinical recommendations was made (Table 3). 
The reported advantages of this method were that often 
the experts identified new evidence before it was available 
in electronic databases, and they therefore proposed that 
the literature search could be done quarterly and limited 
to Medline, the Cochrane library, and meeting proceed- 
ings. The authors described the process as intensive, and 
they foresaw that the updating process would need to 
change if many more guidelines had to be updated. 

Discussion 

Our systematic review shows that the available research 
about the monitoring and updating of CPGs is scarce, 



Table 2 Stages of the strategies* 



Author 
and year 



Assessing the 
need to update 



Scope definition 



Update group 
(number of participants) 



Evidence search strategy 



Evidence selection, 
assessment and synthesis 



Update 

recommendations 



Presentation 
of updates 



Shekelle 
2001 [3] 



Bosquet 
2003 [9]* 



Gartlehner 
2004 [7] 



Nunes 
2009 [15]*§ 



Eccles 
2002 [10] 



Newton 
2006 [14]t§ 



Parmelli 
2010 [16] 



Within specific 
timeframe 



Monitoring 



Within specific 
timeframe 



Within specific 
timeframe 



Within specific 
timeframe 



•IS 



Within specific 
timeframe 



Identification 
of main 

recommendations 



Discussion about 
priority of topics 
to be updated 



Developed analytic 
framework, critical 
key questions and 
eligibility criteria 



Reviewers (2) Chairs of 
the original CPG expert 
panel (15) Members 
related with original 
CPGs (2) Members of the 
original CPG expert panel 
(121) Nonpanel experts (8) 



Methodologist (N5) 
Original working 
group (NS) 



Clinicians (4) Health 
services researchers (2) 
Reviewers (4) Librarians 
(NS) National or 



Survey to national/ international experts (13) 
international experts 



NS 



NS 



Additional research 
questions 



Mail consultation to 
experts from 
development group 
Relevant evidence 



Methodologist (3) 
Clinicians (3) 



General practitioners (8) 
Consultants (2) Nurses (2) 
Health economists (2) 
Guideline methodologist (2) 



Methodologist (9) 
Clinicians (6) 



Coordinating group (10) 3 
multidisciplinary panels (NS) 
External methodological 
group (NS) 



Restricted search 5 key medical 
journals and key specialty journals 
Reviews, editorials, commentaries 
Additional search in NGC and 
web sites of CPGs publishing 
organizations Surveyed experts 
by mail 



Exhaustive search MEDLINE, 
EMBASE, SBU and APM web sites 



Modified Shekelle et al. 
[Shekelle2001] search AIM journals 
and non-AIM speciality journals 
Reviews, editorials, commentaries, 
guidelines Additional search in 
PreMedline, HSTAT, The Cochrane 
Library and selected NIH websites 
Surveyed to national or international 
experts Exhaustive search MEDLINE, 
The Cochrane Library Reviews, 
meta-analysis, RCTs Surveyed to 
national or international experts 

Restricted search Guidelines 



Exhaustive search MEDLINE, EMBASE, 
The Cochrane Library 



Exhaustive search NS 



Exhaustive search MEDLINE, EMABASE, 
central and databases for conference 
(ASCO) SRs, meta-analysis, RCTs 



Reviewed references NA 
(title, abstract and articles) 
Reviewed the relevance of 
selected references Based 
on new evidence and 
judgment guidelines were 
classified in to: major 
update, minor update or 
still valid 



NA 



Selection of references 
Evaluation the impact of 
reference on existing 
guideline: consistent 
with existing guidelines, 
inconsistent or concern 
new topics 

Assessed the relevance of 
abstracts, full-text articles 
and original studies Used 
of update memos with 
studies that judged as 
eligible and addressed 
a critical key question 



Selection and assessment 
guidelines with AGREE 
Narrative summary of the 
methodological aspects 
and major findings of 
each guideline 

Systematic review 
Evidence tables or 
text summaries 



Verification of results and 
study quality from the 
original review 



Evidence selection 
Application GRADE 
methodology Evidence 
tables Informed and 
discussed with panellist 
in meetings 



: ia 



NA 



NA 



MA 



3 situations: new 
recommendations; 
refinement recommendations; 
and unchanged 
recommendations 

Integrating a 
qualitative and 
quantitative data on 
the prior review 
and developing 
recommendations 

Classified strength of 
recommendation in 4 
levels: strong positive, 
weak positive, weak 
negative and strong 
negative Voted by 
experts in meeting 



NA 



NS 



NS 



NS 



Table 2 Stages of the strategies* (Continued) 



Johnston Monitoring 
2003 [8] 



NS 



Health information specialist 
(3) Lead author of the original 
guideline (NS) DSG who 
developed guideline: chair, 
members and research 
assistant (NS) 



Exhaustive search MEDLINE (monthly), 
CancerList (monthly), HealthStar (monthly), 
The Cochrane Library (quarterly) Guidelines, 
SRs, RCTs Additional search in key journals 
and proceedings meeting Notifications 
by DSG members 



Identified potentially 
relevant new trials, 
meta-analyses an 
evidence-based 
guidelines or update 
results from trials 
included in original 
guideline Reviewed 
abstracts and articles 
Considered relevance 
of each reference to 
original guideline question 
Interpreted new evidence 
in the context of original 
guideline Descriptive and 
summaries of new evidence 



DSG choose implications 
of new evidence on clinical 
recommendations: unchanged 
recommendation, strength 
recommendation, change 
recommendation 



Notice linked to each 
guideline to keep 
practitioners aware that 
regular update search 
conduced Update 
bulletin with new 
evidence emerged 
Notice at the top of 
the guideline to alert 
practitioners that 
guideline was review 
Removed obsolete 
guidelines Integrated 
evidence update into 
original report 
(dynamic 'living' 
practice guidelines) 



*Ordered by type of strategy and chronology; tOnly abstract available; § Information provided by the author. 

Abbreviations: AIM Abridged Index Medicus, ARM Agence de Presse Medicale, ASCO American Society of Clinical Oncology, CPG Clinical practice guideline, DSG Disease site group, GRADE Grading of Recommendations 
Assessment, Development and Evaluation, HSTAT Health Service/Technology Assessment Text, NGC National Guideline Clearinghouse, NIH National Institutes of Health, NA Not applicable, NS Not stated, RCT. 
Randomized controlled trials, SBU Swedish Council on Health Technology Assessment, SR Systematic review. 
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Table 3 Results of the strategies* 


Author and year 


Search performance (%) 


Update performance (%) 


Shekelle2001 [3] 


208 articles reviewed/7150 articles initially identified (2,9) 


1 3/1 7 CPGs (76,5) 


Bosquet 2003[9]* 


1 18 references submitted to the working group/261 references initially identified (45,2) 


NS 


Gartlehner 2004 [7] 


Modified Shekelle et al. search: 36 eligible studies/1382 citations initially identified (2,6) 


NS 




Exhaustive search: 45 eligible studies/3687 citations initially identified (1,2) 


NS 


Nunes 2009 [15]*§ 


7 guidelines reviewed/25 guidelines initially identified (28) 


0/NS recommendations 


Eccles 2002 [1 0] 


Angina CPG: 59 acceptable paper/5941 citations initially identified (1) 


0/NS recommendations 




Asthma CPG: 79 acceptable paper/7560 citations initially identified (1) 


0/NS recommendations 


Newton 2006 [14]*§ 


43 studies included/19423 citations initially identified 


1 1/NS questions 


Parmelli 2010 [16] 


24 papers included/686 records initially screened (3,5) 


6/1 5 recommendations (40) 


Johnston 2003 [8] 


19 citations with impact on recommendation/80 citations initially identified (23,8) 


6/20 CPGs (30) 



"Ordered by type of strategy and chronology; tOnly abstract available; information provided by the author. 
Abbreviations: CPG Clinical practice guideline, NS Not stated. 



and little is known at present about the most efficient 
way of keeping guidelines valid. Furthermore, the sub- 
optimal reporting and the wide variability in the design, 
choice of outcomes, type of search strategy, and breadth 
of topics makes the assessment of the included studies 
difficult. 

We identified eight studies with three main different 
objectives: evaluation if guidelines are out of date and, 
hence, how often they need updating [3,7,9,15], and the 
evaluation of updating strategies without [10,14,16] or 
with continuous monitoring new evidence [8]. Regard- 
less of the goal, the included studies followed similar 
stages in their process. However, authors generally did 
not describe the process in enough detail to clearly iden- 
tify the different stages. 

The most detailed phase of the studies was the literature 
search method. Strategies evaluating if CPGs were out of 
date [3,7] used more restricted searches (limited to 
reviews, editorials or commentaries of specific journals 
and expert collaboration) than updating strategies. These 
approaches aim to identify the relevant new evidence 
without performing an exhaustive search; however, they 
risk omitting key references (references that trigger a 
modification of a recommendation). The evidence for 
their performance is so far limited [3,7], with only one 
single head-to-head comparison available between a 
restricted search and an exhaustive search [7]. According 
to the results observed in Gartlehner et al. [7], the evi- 
dence identified by this more restricted search would be 
enough to assess the need to update a CPG. However, it 
remains unclear whether these search results would be a 
sufficient way to actually update recommendations. 

Strategies that aimed to update [10,16] or continuously 
monitor and update CPGs [8] used more exhaustive 
searches and were generally very similar to the ones used 
in the development processes. These exhaustive search 
strategies are more time-consuming than restricted 



searches as the searches trade what is hoped will be higher 
sensitivity for low specificity. However, the study of 
Gartlehner et al. [7] suggests that sensitivity for key stu- 
dies is not increased by an exhaustive search process. 

Although search strategies were specified in the 
included studies, search results were difficult to compare 
across studies due to the variability of the search per- 
formance outcomes (e.g., Shekelleef al. [3] reported the 
articles reviewed and Gartlehner et al. [7] reported just 
the eligible studies). Furthermore, reported results would 
need to be adjusted by the number of CPGs or recom- 
mendations updated and the time to update them. 
Unfortunately, this information was not available for 
most of the studies. 

Only three studies [3,8,16] reported updating perform- 
ance results. For example, Parmelli et al. updated 40% of 
the recommendations about breast, colorectal, and lung 
cancer treatment three years after the development of 
the recommendations [16]. The update process should 
focus on the recommendations, instead of the full guide- 
line, because it could provide an opportunity to make 
the process more efficient. The GRADE system [19], 
already used by Parmelli et al. [16], could provide a 
framework to systematically and explicitly assess to what 
extent the new evidence can modify the different factors 
that ultimately influence the direction and strength of 
the recommendations (quality of the evidence, balance 
between benefits and harms, patients' values and prefer- 
ences, and resource use) [18]. 

Future research studies should standardize outcomes of 
interest. In relation to the search performance, authors 
should report the number of key references (references 
that triggered a modification of a recommendation) from 
the number of references initially identified. In relation to 
the updating performance, measures should include the 
number of recommendations updated from total of the 
CPG recommendations. Finally, regarding resource use, 
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studies should report the number of participants and the 
time spent in the updating process. 

All the studies described the composition of the updat- 
ing working group involved in the process; however, the 
total number of participants and their roles was gener- 
ally unclear. In general, similarly to the development of 
guidelines, there was a core method group and a larger 
group of clinical experts (with a variable degree of 
involvement). Only three studies included the time 
devoted to the process (range eight to ten months) 
[8,14,16]. Nevertheless, this information is difficult to 
generalise because it is highly dependent on the goal of 
the strategy, the scope of the guideline, the methodo- 
logical expertise of the group members or the economic 
resources of the different institutions involved, among 
other possible factors. Had this information been more 
detailed, it could have been used as a guide to assist in 
the development and updating of CPGs programs. 

There is a need to develop more efficient monitoring 
and updating strategies for CPGs and, for this, rigorous 
research is crucial. The gold standard strategy to update 
CPGs should include the identification of new evidence by 
an exhaustive search and the update of the recommenda- 
tions [10,16]. One of the most resource-intensive phases 
where there is more room for improvement is the litera- 
ture search. Other restricted search strategies or resources 
like McMaster Premium Literature Service or Clinical 
Queries, have shown to improve the efficacy of keeping 
systematic reviews up to date [20]. This remains to be 
replicated in guidelines. Other areas to explore are the 
performance of databases (e.g., EMBASE) or the role of 
clinical experts as a source of references, with these more 
limited strategies [21]. 

In relation to survival time, Shekelle et al. [3] proposed, 
as a general rule, to assess the validity of CPGs every three 
years. Regardless of the time frame, which is highly 
dependent on the health topic, it would be desirable to 
develop a dynamic warning system to identify new relevant 
evidence (monitoring) and evaluate the need to update. A 
potential tool that could be easily implemented to monitor 
the new evidence is the automated periodic searches in 
MEDLINE, using their Selective Dissemination of Informa- 
tion service [22]. Complementary, guideline groups could 
monitor publication rate of specific MeSH terms in relation 
with a CPG topic. At the moment this information is not 
readily available in biomedical databases (e.g., MEDLINE). 

Our study has several strengths. We performed an 
exhaustive systematic review including contacting authors 
for additional information. In our review, we proposed a 
novel classification of CPGs updating strategies according 
to their objective: evaluation CPGs obsolescence, updat- 
ing, and continuously monitoring and updating CPGs. 
Finally, we also propose standardised reporting frame- 
work, including outcomes, for research purposes. 



Our study has limitations. First, it is possible that we 
did not identify all studies due to publication bias or to 
the omission of some more specialised information 
sources (e.g., conference proceedings). Second, the dif- 
ficulty of synthesizing, evaluating, and comparing complex 
methodological studies, without a standardized reporting, 
as opposed to systematic reviews [23] or comparative ef- 
fectiveness reviews [24], makes the analysis and interpre- 
tation of results challenging. 

Conclusions 

There is limited evidence about the optimal strategies 
for monitoring and updating CPGs. A restricted search 
is likely to be sufficient to monitor new evidence and as- 
sess the need to update; however, more information is 
needed about the timing and type of search. Only the 
exhaustive search strategy has been assessed for the 
update of CPGs. 

Our review highlights suboptimal reporting, and wide 
variability in the design, choice of outcomes, and type of 
search strategy of the available studies. The development 
and evaluation of more efficient strategies is needed to 
improve the timeliness and reduce the burden of 
maintaining the validity of CPGs. Future research 
studies should adequately report their methods and 
results. 
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